Quality and sustainability of software tools in neuroscience

In computational neuroscience, the continual need for new software tools has led to the creation of software in a disparate and ad-hoc fashion with considerable overlap but little compatibility between different tools. Much of the resulting software is made freely available, but it is not always clear that it is sufficiently mature in terms of domain coverage, validity, documentation or usability, to be useful to other researchers. Software tools for scientific use should meet certain minimal conditions of correctness, usability and reliability. For databases and tools that are made publicly available, additional criteria of maintainability and sustainability should be met. We focused on 36 neuroscience software tools and asked how the culture and practice of software development affects the validity and trustworthiness of scientific software. We identified two sets of categories: one for how software is developed, and one for the end products of such work. The development models include pet projects driven by single individuals or small groups, collaborative projects, and projects that are driven with up-front funding. The end products can be classified as proof of concept work, private tools and public tools according to their accessibility and state of readiness for scientific work. Based on our findings we suggest ways in which the current practice of software development in computational neuroscience can be improved along with checklists for developers, reviewers, and scientists to help assess the quality and research readiness of a particular piece of software.